Fail a rolling restart job that did not restart every node - #3498

test:unit:main hit its 50-minute timeout in the components suite (interrupted activation recovery), with failures in ComponentV1.test.js and componentPreparationLock.test.js. Both files pass in isolation on this branch and on main (59/59 each), and neither touches changed code.

QA-PR harper#3498 — Does a real two-node rolling restart report ERROR, with per-peer outcomes, when the peer cannot restart?

Status: pending

source:pr:3498 · harper b323922 · branch fix/rolling-restart-job-outcome · base main · [two-node harper-pro cluster built against this core]

Assumption under test: A replicated restart_service ({"operation":"restart_service","service":"http","replicated":true}) ends COMPLETE only when every peer's own restart job completed; otherwise get_job shows status: ERROR, a message naming the peer, and result.replicated with one entry per peer ({ node, message } restarted, { node, error } not). The origin classifies peer failures from the text and socket code the real replicator delivers (sendOperationToNode): a write-queue back-pressure refusal ("Outstanding write transactions have too long a queue") is retried after the other peers; "outcome there is unknown" (timeout / connection closed first) and coded socket errors other than ECONNREFUSED/ENOTFOUND/EAI_AGAIN/EHOSTUNREACH/ENETUNREACH halt the roll.

Not covered by the PR's tests: the restart tests stub server.replication.sendOperationToNode with hand-written error strings and never runs jobProcess → updateJob → get_job. Nothing exercises the real replicator's error text/code, the persisted hdb_job row on ERROR, or a real peer job insert failing.

Question: Build harper-pro with its core submodule at the PR head (git -C core fetch origin pull/3498/head && git -C core checkout FETCH_HEAD, then build). Start two nodes A and B, cluster them (add_node), confirm replication is connected. Arms, each started from A and polled with get_job on A until terminal:

1. Baseline: both nodes healthy → expect COMPLETE, result.replicated = one { node: B, message } entry.
2. Peer down: stop B's process entirely → expect ERROR naming B with an ECONNREFUSED-class error in result.replicated (the roll does not need to halt; there is only one peer).
3. Peer refuses: on B, make the job-row insert fail (e.g. storage.maxTransactionQueueTime set very low on B plus a sustained write load on B, or any reproducible way to make B's hdb_job insert throw) → expect A's job to retry B (log line "was refused ... retrying after the other nodes") and end ERROR or COMPLETE consistent with whether B eventually accepted; B's log must not contain "Empty job passed to parseMessage".
4. Peer restarts mid-poll: with a 3rd node C if cheap, restart B's workers externally while A polls B's job → no unhandledRejection lines in A's job-thread log, job still settles.
5. In every arm, check get_job on A returns result.replicated on ERROR (persisted, not only in memory).

Oracle: arm 1 COMPLETE; arm 2 ERROR with B named in message and a { node: "B", error: ... } entry in result.replicated; arm 3 COMPLETE if B accepted a retry, otherwise ERROR the same way; no "Empty job passed to parseMessage" anywhere; zero unhandledRejection lines on A in any arm. A COMPLETE in arm 2 (or in arm 3 without B's restart), a missing result.replicated on ERROR, or an unhandled rejection is a finding.

Regression A/B: on failure, repeat with core at the PR's merge base on main (d4f9fb9); expected there: arm 2 reports COMPLETE with B's failure buried in result.replicated (the reported bug), arm 3 shows "Empty job passed to parseMessage".

Axes: replication (operation forwarding), jobs lifecycle (hdb_job persistence), restart lifecycle, mixed failure classes (refused / unreachable / dropped mid-poll); storage RocksDB.

Fix harper#3468 on main: a replicated restart_service job (the job behind deploy_component with restart: "rolling") reports status COMPLETE even when a node was never restarted, and a failed job insert on a peer is reported as the misleading "Empty job passed to parseMessage". Address all four defects listed in the issue: (1) bin/restart.ts pushes { node, message } and continues when sendOperationToNode throws, then returns { replicated }, so the job completes normally; make the job end in ERROR naming every node that failed or was skipped, keeping the per-node result.replicated entries. (2) server/jobs/jobs.ts addJob catches the insert error and returns { success: false } with no createdJob; executeJob (server/serverHelpers/serverUtilities.ts) only checks if (result) and builds a RunnerMessage with job undefined, so parseMessage throws. Make executeJob check success/createdJob and surface the original error (e.g. the write-queue back-pressure error). (3) Retry a node whose failure is a transient back-pressure error (the 'Outstanding write transactions have too long of queue, please try again later' class) a bounded number of times, for example re-queueing it after the remaining nodes, before marking it failed. (4) The get_job poll in bin/restart.ts runs in setInterval(async () => ...) with no try/catch, producing unhandledRejection (Connection closed 1006 / socket disconnected before TLS established) while a peer restarts; handle those and keep polling.

Acceptance

Unit tests: a restart job with one peer whose operation throws ends ERROR and names that peer; a peer whose job insert fails surfaces the original insert error from executeJob, not 'Empty job passed to parseMessage'; a transient back-pressure failure that succeeds on retry leaves the job COMPLETE; a peer connection drop during the get_job poll produces no unhandled rejection. Each new test fails on unmodified main. Existing restart/deploy unit and integration suites pass. Do not change deploy_component's own response or the deploy record (harper#3058/#3060 scope). Draft PR only.